Skip to content

feat(cognitive): improve historian quality and stream LLM metrics - #92

Merged
69gg merged 9 commits into
mainfrom
feature/better-historian
Aug 13, 2026
Merged

feat(cognitive): improve historian quality and stream LLM metrics#92
69gg merged 9 commits into
mainfrom
feature/better-historian

Conversation

@69gg

@69gg 69gg commented Aug 13, 2026

Copy link
Copy Markdown
Owner

Summary

  • 史官侧写合并改为最新事实优先,并按提示词克制扩写、合并去冗;事件改写尽量提炼为带时间锚点的独立事实。
  • 收紧 end.observations:只写值得日后检索的写实内容,宁缺毋滥,禁止流程决策、否定清单和碎碎念。
  • 流式 LLM 调用记录 TTFT 与 TPS(非流式只记总耗时),版本同步至 3.11.1。

Test plan

  • 跑一轮带侧写更新的对话,确认冲突时以当前批次覆盖过时特征,侧写不会无限膨胀
  • 确认史官改写把“谁说了什么”尽量变成带时间锚点的独立事实,无法核实时保留言说行为
  • 确认无实质事实时 end.observations 为空,不会硬凑静默决策/否定清单
  • 流式调用日志与 data/token_usage.jsonlttft_seconds / tokens_per_second;非流式记录不含这两项
  • uv run pytest tests/test_cognitive_historian.py tests/test_llm_streaming.py tests/test_token_usage_unit.py tests/test_system_prompt_constraints.py

Made with Cursor

Summary by CodeRabbit

  • New Features

    • Streaming model calls now report time-to-first-token and tokens-per-second when available.
    • Usage statistics include additional streaming performance metrics.
  • Improvements

    • Memory observations now prioritize concise, factual, searchable information and remain empty when no meaningful facts exist.
    • Profile updates better reflect current information, reduce duplication, and preserve relevant time context.
    • Event summaries are more independent, time-anchored, and resistant to unnecessary detail.
  • Documentation

    • Updated guidance for memory behavior, usage statistics, and streaming metrics.
  • Release

    • Updated to version 3.11.1.

69gg and others added 7 commits August 11, 2026 10:12
Inject merge-time now and profile updated_at so the historian can resolve stale traits against the current input batch.

Co-authored-by: Cursor <cursoragent@cursor.com>
Guide lightweight rewrite to prefer verifiable fact sentences with absolute time anchors over speech-act shells.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@coderabbitai

coderabbitai Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Warning

Review limit reached

@69gg, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 29 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 0eeae5d6-caca-44bf-84ff-0fe07b66ea83

📥 Commits

Reviewing files that changed from the base of the PR and between 1e77693 and 12fbd24.

📒 Files selected for processing (1)
  • tests/test_cognitive_historian.py
📝 Walkthrough

Walkthrough

The change tightens cognitive-memory observation and historian profile rules. It adds provider-aware streaming TTFT/TPS measurement and optional token-usage persistence. Release metadata and documentation now describe version 3.11.1 and these behaviors.

Changes

Historian and streaming metrics

Layer / File(s) Summary
Observation and historian processing
AGENTS.md, CLAUDE.md, README.md, docs/cognitive-memory.md, docs/message-batching.md, res/prompts/*, src/Undefined/cognitive/historian/*, src/Undefined/skills/tools/end/*, tests/test_cognitive_historian.py, tests/test_system_prompt_constraints.py
Observation guidance now requires substantive current-batch facts and permits empty arrays. Historian rewriting and profile merging now use time-aware, latest-first, deduplicated facts.
Streaming timing and usage persistence
src/Undefined/ai/llm/*, src/Undefined/token_usage_storage.py, tests/test_llm_streaming.py, tests/test_token_usage_unit.py, docs/configuration.md, docs/model-compatibility.md, docs/usage.md, ARCHITECTURE.md
Streaming requests detect first meaningful output, calculate TTFT/TPS, log available metrics, and persist optional values.
3.11.1 release metadata
CHANGELOG.md, pyproject.toml, src/Undefined/__init__.py, apps/undefined-chat/..., apps/undefined-console/...
Project and application versions change from 3.11.0 to 3.11.1. The changelog adds the release entry.

Estimated code review effort: 4 (Complex) | ~45 minutes

Mergeability Score: 🔵 Low · up to 1e776

Profile merging now prioritizes newer facts, but timezone-less timestamps may be mixed with timezone-aware values, making recency selection ambiguous; clarify or normalize this behavior before merging.

Sequence Diagram(s)

sequenceDiagram
  participant Requester
  participant StreamingProvider
  participant TokenUsageStorage
  Requester->>StreamingProvider: Start streaming request
  StreamingProvider-->>Requester: Return first meaningful output event
  Requester->>Requester: Calculate TTFT and TPS
  Requester->>TokenUsageStorage: Persist available metrics
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 17.14% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main changes: improved historian quality and added streaming LLM metrics.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feature/better-historian

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@69gg
69gg marked this pull request as ready for review August 13, 2026 01:35

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 8

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/message-batching.md`:
- Line 46: Define one consistent observation provenance contract across
docs/message-batching.md:46-46, res/prompts/undefined.xml:240-252 and :961-965,
and res/prompts/undefined_nagaagent.xml:240-252 and :1020-1024. Decide whether
verified assistant-action outcomes are valid observation sources, document that
decision, and update each end.observations and memory-management prompt to
enforce the same rule; preserve the requirement that observations are otherwise
limited to substantive facts from the current input batch and use an empty array
when none exist.

In `@res/prompts/historian_profile_merge.md`:
- Around line 118-135: Update the fenced code block in the positive example to
specify the text language on its opening fence, resolving the MD040 markdownlint
violation while preserving the example content.

In `@res/prompts/historian_rewrite.md`:
- Around line 7-21: Update the examples in the “轻量独立事实” section to use the
required format “QQ号123(昵称张三)” instead of “张三(123)”. Also revise the name-format
rule around line 25 to require cognitive observations to use
“QQ号<user_id>(昵称<name>)”, preserving the explicit numeric identifier.

In `@src/Undefined/ai/llm/streaming.py`:
- Around line 246-266: Update responses_event_marks_ttft so
response.function_call_arguments.delta events return
bool(stringify_stream_delta(event.get("delta"))) instead of unconditionally
marking TTFT. Add a regression test covering an event with an empty string delta
and assert it is not treated as TTFT.

In `@src/Undefined/cognitive/historian/worker.py`:
- Around line 680-705: Update the timestamp setup around now_local_dt,
now_utc_dt, and timezone_label to resolve job["timezone"] with ZoneInfo, using a
safe fallback for invalid or missing values. Derive local and UTC timestamps
from the same current instant, ensure the prompt’s now_local uses the resolved
job timezone, and emit future updated_at values with an explicit offset; apply
the same correction to the corresponding logic near the profile-merge handling
at lines 732-736.

In `@src/Undefined/skills/tools/end/config.json`:
- Line 16: Require user-centered cognitive-memory observations to identify users
as QQ号<sender_id>(昵称<name>), updating the observations description in
src/Undefined/skills/tools/end/config.json:16-16 and the function description at
src/Undefined/skills/tools/end/config.json:5-5. Mirror this stable-ID
requirement in docs/cognitive-memory.md:71-71 and docs/cognitive-memory.md:8-8,
and align src/Undefined/skills/tools/end/README.md:7-7. Extend
tests/test_system_prompt_constraints.py:448-455 and
tests/test_system_prompt_constraints.py:492-498 to assert the rule in both
prompt and end-tool schema contracts.

Apply the same fix in `@docs/cognitive-memory.md` at line 8.

Apply the same fix in `@tests/test_system_prompt_constraints.py` around lines 448
- 455.

Apply the same fix in `@src/Undefined/skills/tools/end/README.md` at line 7.

In `@src/Undefined/token_usage_storage.py`:
- Around line 83-89: Update to_optional_float to return None when the parsed
float is non-finite or negative, while preserving None for null and invalid
inputs and valid non-negative finite values. Add tests covering "NaN",
"Infinity", and negative values.

In `@tests/test_cognitive_historian.py`:
- Around line 375-385: Update the merge template contract test assertions around
the existing now_local check to also require the {now_utc} placeholder and
verify the generated prompt includes a valid UTC timestamp value. Preserve the
existing assertions and add coverage that would fail if the worker stops
supplying or rendering now_utc.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 91f3569e-3c22-43ce-b4f1-30b63193ca66

📥 Commits

Reviewing files that changed from the base of the PR and between 5a8d3ae and 86b0324.

⛔ Files ignored due to path filters (5)
  • apps/undefined-chat/package-lock.json is excluded by !**/package-lock.json
  • apps/undefined-chat/src-tauri/Cargo.lock is excluded by !**/*.lock
  • apps/undefined-console/package-lock.json is excluded by !**/package-lock.json
  • apps/undefined-console/src-tauri/Cargo.lock is excluded by !**/*.lock
  • uv.lock is excluded by !**/*.lock
📒 Files selected for processing (33)
  • AGENTS.md
  • ARCHITECTURE.md
  • CHANGELOG.md
  • CLAUDE.md
  • README.md
  • apps/undefined-chat/package.json
  • apps/undefined-chat/src-tauri/Cargo.toml
  • apps/undefined-chat/src-tauri/tauri.conf.json
  • apps/undefined-console/package.json
  • apps/undefined-console/src-tauri/Cargo.toml
  • apps/undefined-console/src-tauri/tauri.conf.json
  • docs/cognitive-memory.md
  • docs/configuration.md
  • docs/message-batching.md
  • docs/model-compatibility.md
  • docs/usage.md
  • pyproject.toml
  • res/prompts/historian_profile_merge.md
  • res/prompts/historian_rewrite.md
  • res/prompts/undefined.xml
  • res/prompts/undefined_nagaagent.xml
  • src/Undefined/__init__.py
  • src/Undefined/ai/llm/requester.py
  • src/Undefined/ai/llm/streaming.py
  • src/Undefined/cognitive/historian/helpers.py
  • src/Undefined/cognitive/historian/worker.py
  • src/Undefined/skills/tools/end/README.md
  • src/Undefined/skills/tools/end/config.json
  • src/Undefined/token_usage_storage.py
  • tests/test_cognitive_historian.py
  • tests/test_llm_streaming.py
  • tests/test_system_prompt_constraints.py
  • tests/test_token_usage_unit.py

Comment thread docs/message-batching.md Outdated
Comment thread res/prompts/historian_profile_merge.md
Comment thread res/prompts/historian_rewrite.md
Comment thread src/Undefined/ai/llm/streaming.py
Comment thread src/Undefined/cognitive/historian/worker.py Outdated
Comment thread src/Undefined/skills/tools/end/config.json Outdated
Comment thread src/Undefined/token_usage_storage.py
Comment thread tests/test_cognitive_historian.py
Keep observation provenance, QQ-ID naming, job timezone offsets, and
TTFT/TPS parsing consistent so merge prompts and usage logs stay trustworthy.

Co-authored-by: Cursor <cursoragent@cursor.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
tests/test_cognitive_historian.py (1)

403-411: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Cover malformed and non-dictionary frontmatter.

The helper now promises empty results for malformed YAML and non-dictionary YAML. These tests cover missing frontmatter and a missing updated_at field, but not those two fallback paths. Add both cases.

Based on learnings, add or update test coverage for behavior changes in cognitive memory.

Suggested cases
+    assert _extract_frontmatter_updated_at("---\n- item\n---") == ""
+    assert _extract_frontmatter_updated_at("---\nupdated_at: [\n---") == ""
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/test_cognitive_historian.py` around lines 403 - 411, Add cases to
test_extract_frontmatter_updated_at for malformed YAML and valid YAML whose
top-level value is not a dictionary, asserting both return an empty string. Keep
the existing timestamp, missing-field, and absent-frontmatter assertions
unchanged.

Source: Learnings

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/test_cognitive_historian.py`:
- Around line 423-428: Update test_now_in_job_timezone_invalid_falls_back to
compare now_local’s UTC offset and timezone name against
datetime.now().astimezone(), asserting the invalid timezone falls back to the
system-local timezone rather than merely checking the label differs. Retain the
existing UTC and consistency assertions.

---

Outside diff comments:
In `@tests/test_cognitive_historian.py`:
- Around line 403-411: Add cases to test_extract_frontmatter_updated_at for
malformed YAML and valid YAML whose top-level value is not a dictionary,
asserting both return an empty string. Keep the existing timestamp,
missing-field, and absent-frontmatter assertions unchanged.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 3e4c180b-fd8b-4160-9ca1-15ebc4aa9470

📥 Commits

Reviewing files that changed from the base of the PR and between 86b0324 and 1e77693.

📒 Files selected for processing (16)
  • docs/cognitive-memory.md
  • docs/message-batching.md
  • res/prompts/historian_profile_merge.md
  • res/prompts/historian_rewrite.md
  • res/prompts/undefined.xml
  • res/prompts/undefined_nagaagent.xml
  • src/Undefined/ai/llm/streaming.py
  • src/Undefined/cognitive/historian/helpers.py
  • src/Undefined/cognitive/historian/worker.py
  • src/Undefined/skills/tools/end/README.md
  • src/Undefined/skills/tools/end/config.json
  • src/Undefined/token_usage_storage.py
  • tests/test_cognitive_historian.py
  • tests/test_llm_streaming.py
  • tests/test_system_prompt_constraints.py
  • tests/test_token_usage_unit.py
💤 Files with no reviewable changes (1)
  • src/Undefined/ai/llm/streaming.py
🚧 Files skipped from review as they are similar to previous changes (12)
  • res/prompts/undefined.xml
  • src/Undefined/skills/tools/end/config.json
  • tests/test_token_usage_unit.py
  • src/Undefined/skills/tools/end/README.md
  • tests/test_system_prompt_constraints.py
  • docs/cognitive-memory.md
  • src/Undefined/cognitive/historian/worker.py
  • tests/test_llm_streaming.py
  • res/prompts/historian_profile_merge.md
  • res/prompts/undefined_nagaagent.xml
  • src/Undefined/token_usage_storage.py
  • res/prompts/historian_rewrite.md

Comment thread tests/test_cognitive_historian.py
Assert invalid job timezones fall back to the system-local zone, and cover
malformed or non-mapping YAML when reading profile updated_at.

Co-authored-by: Cursor <cursoragent@cursor.com>
@69gg
69gg merged commit c59f296 into main Aug 13, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant